The present application relates to a method of compensating an imager-created image of a photographing object for a distortion caused by a hand movement or the like at a time to take the image of the photographing object with an optical zoom operation carried out and an image compensation apparatus adopting the method of compensating such a taken image as well as relates to an image taking method and an image taking apparatus. The imager-created image is an image created on an image taking device employed in the image compensation apparatus or the image taking apparatus as an imager.
If a moving picture is taken by making use of an electronic image taking apparatus, which stores an electronic imager-created image of a photographing object in an image taking device employed in the electronic image taking apparatus and reads out the image later from the image taking device also referred to hereafter as an imager, due to the so-called hand movement, the image taking device employed in the image taking apparatus held by the hands of the photographer may physically move to result in a change in position at a relatively high velocity in the horizontal and/or vertical directions of the image of the photographing object so that the change in position appears in the imager-created image as an image distortion such as an image trembling. Examples of such an electronic image taking apparatus are a video camera and a digital still camera.
That is to say, without a hand movement, a location at which the imager-created image is formed on the image taking face of the image taking device is a fixed location. Thus, if a plurality of frame images is arranged in the time-axis direction, the frame images overlap each other at exactly the same location as shown in FIG. 48A. If a hand moves, however, the location at which the imager-created image is formed on the image taking face of the image taking device is not a fixed position. Thus, if a plurality of frame images is arranged in the time-axis direction, the frame images swing from location to location as shown in FIG. 48B.
If a zoom lens is employed in a telescope, in particular, the phenomenon caused by a hand movement as described above occurs in a striking manner most likely. The occurrence of such a hand-movement phenomenon raises a problem that a still observation object inevitably looks like a moving object so that the image of the object is difficult to see. In addition, a hand movement also causes a phenomenon in which an imager-created image of the photographing object is generated in a defocused state.
On top of that, in an attempt to obtain an imager-created image output from an image-taking process by superposing a plurality of frame images on each other in an operation to take a picture of a still face of a photographing object, as described above, the imager created image of the photographing object is distorted due to a defocused state.
As techniques to compensate an image for a distortion caused by a hand movement, there have been proposed an optical hand-movement compensation method making use of a sensor for detecting a hand movement and a sensorless hand-movement compensation method by which a digital signal process is carried out on the imager-created image in order to detect a hand movement and compensate the image for a distortion caused by the hand movement.
The method adopted by a consumer apparatus currently available in the market as a method for compensating a still picture for a distortion caused by a hand movement is the aforementioned optical hand-movement compensation method making use of a gyro sensor (or an acceleration sensor) for measuring a hand-movement displacement vector. The measured hand-movement displacement vector is fed back to a mechanism in high-speed control to prevent an image projected on an image sensor such as the CCD (Charge Coupled Device) or the CMOS (Complementary Metal Oxide Semiconductor) imager from being affected by the hand movement.
As the mechanism cited above, there has been proposed a mechanism including a lens, a prism and the imager (or an integrated module including the imager). In the following description, the lens, the prism and the imager are referred to as a lens shift, a prism shift and an imager shift respectively.
In accordance with the sensorless hand-movement compensation method, on the other hand, as disclosed in documents such as Patent Documents 1 and 2, a movement vector of an imager-created image for a screen unit is detected from data of the imager-created image read out from an image taking device used as an imager and, on the basis of the movement vector, the read position of the imager-created image stored in the image taking device is shifted in order to compensate the data for a distortion caused by a hand movement. In this specification, Japanese Patent No. 3,303,312 is referred to as Patent Document 1 whereas Japanese Patent Laid-Open No. Hei 6-86149 is referred to as Patent Document 2.
Patent Document 3 and some other documents have each proposed a technique for compensating a still picture for a distortion caused by a hand movement without making use of a sensor. In this specification, Japanese Patent Laid-Open No. Hei 7-283,999 is referred to as Patent Document 3. A technique proposed in Patent Document 3 adopts an algorithm, in accordance with which, some still pictures are taken consecutively during such a short exposure period that the hand is not moving during the period, and hand-movement displacement vectors between the still pictures are found. Then, in accordance with the hand-movement displacement vectors, the still pictures taken consecutively are added to each other (or averaged) while being shifted in parallel (and rotating in the roll-axis direction) in order to find a final high-quality still picture with no effects of a hand movement and no low-illumination noises.
In general, a video camera has an optical zoom operation unit such as a zoom switch in a configuration allowing the photographer to carry out an optical zoom operation by operating the optical zoom operation unit in order to execute optical zoom control while photographing a moving picture.
In this case, the size of an imager-created image being subjected to an optical zoom operation to, for example, enlarge the image gradually increases with the lapse of time from a start position, which coincides with the center of the image, as shown in FIG. 49.
If the hand of the photographer moves in a photographing operation accompanied by this optical zoom operation, however, the start position from which the size of the image is increasing is inevitably shifted from the center of the image as shown in FIG. 50 so that it is feared that the image is distorted.
If the image can be compensated for the distortion occurring in an optical zoom operation, the effect of the distortion can be lessened. In a process to compensate the image for the distortion by making use of a gyro sensor like the one described before, however, the optical lens and the prism or the image taking device are controlled to move in a direction perpendicular to the optical axis of the incoming light while the zoom control moves the optical lens in a direction parallel to the optical axis. Thus, the control becomes very complicated. As a result, the process to optically compensate an image for a distortion caused by a hand movement was stopped during an optical zoom operation in the past.
In addition, in the case of the sensorless hand-movement compensation method, an image movement vector component caused by an optical zoom operation is added to a hand-movement displacement vector detected from data read out from the image taking device as the data of an imager-created image so that, even if the image is compensated for a distortion on the basis of the detected hand-movement displacement vector, the distortion caused by a hand movement remains in the image. For this reason, also in the case of the sensorless hand-movement compensation method, the process to optically compensate an image for a distortion caused by a hand movement is stopped during an optical zoom operation.
As described above, the image taking apparatus in the related art does not compensate an image for a distortion generated in an optical zoom operation. Thus, if the hand moves in an optical zoom operation, a distortion generated in an optical zoom operation due to a hand movement like the one shown in FIG. 50 remains in the image. The distortion caused by a hand movement in an optical zoom operation is seen in a relatively less striking manner in the case of a small monitor screen. As the degree of image fineness has been rising and the user enjoys more a picture appearing on a large screen in recent years, however, a distortion caused by a hand movement in an optical zoom operation can no longer be ignored.